Vehicle control device, vehicle control method, and storage medium

ABSTRACT

A vehicle control device according to an embodiment includes an imager configured to image surroundings of a host vehicle, a recognizer configured to recognize a surroundings situation of the host vehicle, a driving controller configured to control one or both of speed and steering of the host vehicle on the basis of a result of the recognition of the recognizer, and a controller configured to control the driving controller on the basis of imaging content of the imager, and the controller sets relative position information with respect to the host vehicle for an object around the host vehicle present on a two-dimensional image captured by the imager, and corrects the recognition result of the recognizer on the basis of the set relative position information.

CROSS-REFERENCE TO RELATED APPLICATION

Priority is claimed on Japanese Patent Application No. 2021-167280,filed Oct. 12, 2021, the content of which is incorporated herein byreference.

BACKGROUND Field of the Invention

The present invention relates to a vehicle control device, a vehiclecontrol method, and a storage medium.

Description of Related Art

In recent years, with the aim of providing a sustainable transportationsystem by improving traffic safety or convenience, research has beenconducted on automatically controlling the traveling of vehicles thattransport occupants on the basis of a result of recognizing asurroundings situation. In relation to this, in the field of recognizingsurroundings of a vehicle, a technology for extracting feature pointsfrom images of a road surface of a road captured by a plurality ofrespective cameras, transforming coordinates of the extracted featurepoints into a bird's-eye view coordinate system common to the pluralityof cameras, and determining types of demarcation lines on the basis of aresult of determining a state transition of the feature points extractedfrom the image captured by a camera determined to be available among theplurality of cameras is known (for example, WO2018/109999).

SUMMARY

However, in the related technology, since each of a plurality of imagescaptured by a plurality of cameras is coordinate-transformed into abird's-eye view coordinate system, a positional deviation of demarcationlines or objects such as other vehicles in the image occurs, such that apositional relationship between the demarcation lines and the objectscannot be accurately acquired and traffic safety cannot be improved insome cases.

Aspects of the present invention have been made in consideration of suchcircumstances, and one object of the present invention is to provide avehicle control device, a vehicle control method, and a storage mediumcapable of more accurately recognizing a position of an object includedin an image to perform driving control of a host vehicle, and furtherimproving traffic safety.

The vehicle control device, vehicle control method, and storage mediumaccording to the present invention adopt the following configuration.

(1): A vehicle control device according to an aspect of the presentinvention is a vehicle control device including: an imager configured toimage surroundings of a host vehicle; a recognizer configured torecognize a surroundings situation of the host vehicle; a drivingcontroller configured to control one or both of speed and steering ofthe host vehicle on the basis of a result of the recognition of therecognizer; and a controller configured to control the drivingcontroller on the basis of imaging content of the imager, wherein thecontroller sets relative position information with respect to the hostvehicle for an object around the host vehicle present on atwo-dimensional image captured by the imager, and corrects therecognition result of the recognizer on the basis of the set relativeposition information.(2): In the aspect (1), when the driving controller executes trackingtraveling in which the host vehicle tracks a preceding vehicle and therecognizer recognizes a new tracking target vehicle for the host vehicleor a cut-in vehicle cutting into the host lane in which the host vehicletravels, the controller corrects control content of the drivingcontroller for the tracking target vehicle or the cut-in vehicle on thebasis of the relative position information with respect to the trackingtarget vehicle or the cut-in vehicle.(3): In the aspect (2), when a determination is made that the trackingtarget vehicle and the cut-in vehicle are not present in the host laneon the basis of the relative position information even in a case inwhich the recognizer recognizes the tracking target vehicle or thecut-in vehicle, the controller excludes the tracking target vehicle andthe cut-in vehicle from targets of driving control of the drivingcontroller.(4): In the aspect (2), the controller causes the driving controller toexecute driving control for the tracking target vehicle or the cut-invehicle on the basis of the corrected recognition result.(5): A vehicle control method according to an aspect of the presentinvention is a vehicle control method including, on a computer: imaging,by an imager, surroundings of a host vehicle; recognizing a surroundingssituation of the host vehicle; executing driving control to control oneor both of speed and steering of the host vehicle on the basis of aresult of the recognition; controlling the driving controller on thebasis of imaging content of the imager; and setting relative positioninformation with respect to the host vehicle for an object around thehost vehicle present on a two-dimensional image captured by the imager,and correcting the recognition result on the basis of the set relativeposition information.(6): A storage medium according to an aspect of the present invention isa computer-readable non-transitory storage medium having a programstored therein, the program causing a computer to: image surroundings ofa host vehicle using an imager; recognize a surroundings situation ofthe host vehicle; execute driving control to control one or both ofspeed and steering of the host vehicle on the basis of a result of therecognition; control the driving controller on the basis of imagingcontent of the imager; and set relative position information withrespect to the host vehicle for an object around the host vehiclepresent on a two-dimensional image captured by the imager, and correctthe recognition result on the basis of the set relative positioninformation.

According to the aspects (1) to (6) above, it is possible to moreaccurately recognize a position of an object included in an image toperform driving control of a host vehicle, and further improve trafficsafety.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example of a configuration of avehicle system having a vehicle control device according to anembodiment mounted therein.

FIG. 2 is a diagram illustrating an example of control that is executedby a driving control device.

FIG. 3 is a flowchart illustrating an example of a flow of processingthat is executed by the driving control device.

FIG. 4 is a diagram illustrating an example of a surroundings situationof a host vehicle.

FIG. 5 is a diagram illustrating an area surrounding a contour ofanother vehicle included in a two-dimensional image.

FIG. 6 is a diagram illustrating extraction of demarcation lines.

FIG. 7 is a diagram illustrating removal of noise of a demarcation line.

FIG. 8 is a diagram illustrating extraction of an inclination of thedemarcation line.

FIG. 9 is a diagram illustrating acquisition of left and right edges ofanother vehicle.

FIG. 10 is a diagram illustrating acquisition of edges of the othervehicle using segmentation areas.

FIG. 11 is a diagram illustrating a determination of a lane to whichanother vehicle belongs.

FIG. 12 is a flowchart illustrating an example of a flow of processingfor determining a belonging lane.

FIG. 13 is a diagram illustrating a plurality of determinationconditions.

FIG. 14 is a diagram illustrating content of processing of steps S212 toS214.

FIG. 15 is a diagram illustrating processing for determining an absencesettlement lane.

FIG. 16 is a diagram illustrating an example of information in whichflags are set.

FIG. 17 is a diagram illustrating a first determination pattern.

FIG. 18 is a diagram illustrating a second determination pattern.

FIG. 19 is a diagram illustrating a third determination pattern.

FIG. 20 is a diagram illustrating a fourth determination pattern.

FIG. 21 is a diagram illustrating a second scene in which a part of eachof both demarcation lines cannot be recognized.

FIG. 22 is a diagram illustrating a function of the target capturer.

FIG. 23 is a diagram illustrating a setting of a control transitionratio in a switching period of a target vehicle.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of a vehicle control device, a vehicle controlmethod, and a storage medium of the present invention will be describedwith reference to the drawings. It is assumed that the vehicle controldevice of the embodiment is mounted in a vehicle. The vehicle is, forexample, a vehicle such as a four-wheeled vehicle, and a driving sourcethereof includes an internal combustion engine such as a diesel engineor a gasoline engine, an electric motor, or a combination thereof. Theelectric motor operates using power generated by a power generatorconnected to the internal combustion engine or discharge power of asecondary battery or a fuel cell.

[Overall Configuration]

FIG. 1 is a diagram illustrating an example of a configuration of avehicle system 1 equipped with a vehicle control device according to anembodiment. A vehicle system 1 illustrated in FIG. 1 includes, forexample, a first camera 10, a radar device 12, a second camera 20, ahuman machine interface (HMI) 30, a vehicle sensor 40, a drivingoperator 80, and a traveling force output device 92, a brake device 94,a steering device 96, and a driving control device 100. These devices orequipment are connected to each other by a multiplex communication linesuch as a controller area network (CAN) communication line, a serialcommunication line, a wireless communication network, or the like. Theconfiguration illustrated in FIG. 1 is merely an example, and a part ofthe configuration may be omitted or other configurations may be added.The second camera 20 is an example of an “imager”. The imager mayinclude the first camera 10. The second camera 20 and the drivingcontrol device 100 are examples of a “vehicle control device.”

The first camera 10 is, for example, a digital camera using asolid-state imaging device such as a charge coupled device (CCD) or acomplementary metal oxide semiconductor (CMOS). One or a plurality offirst cameras 10 are attached to any location on a vehicle (hereinafter,a host vehicle M) in which the vehicle system 1 is mounted. For example,when a forward side of the host vehicle M is imaged, the first camera 10is attached to, for example, an upper portion of a front windshield, arear surface of a rearview mirror, or the like. When a backward side ofthe host vehicle M is imaged, the first camera 10 is attached to anupper portion of a rear windshield, a back door, or the like. When asideward side and a rear sideward side of the host vehicle M are imaged,the first camera 10 is attached to a door mirror or the like. The firstcamera 10, for example, periodically and repeatedly images surroundingsof the host vehicle M. The first camera 10 may be a stereo camera.

The first camera 10 further includes a fisheye camera capable of imagingthe surroundings of the host vehicle M at a wide angle (for example, at360 degrees). The fisheye camera is attached, for example, to an upperportion of the host vehicle M and images the surroundings of the hostvehicle M at a wide angle in a horizontal direction. The fisheye cameramay be realized by combining a plurality of cameras (a plurality ofcameras that image a range of 120 degrees or a range of 60 degrees inthe horizontal direction).

The radar device 12 radiates radio waves such as millimeter waves to thesurroundings of the host vehicle M and detects radio waves (reflectedwaves) reflected by an object in the surroundings to detect at least aposition (a distance and orientation) of the object. One or a pluralityof radar devices 12 are attached to arbitrary locations on the hostvehicle M. The radar device 12 may detect a position and speed of theobject in the surroundings using a frequency modulated continuous wave(FM-CW) scheme.

The second camera 20, for example, a digital camera using a solid-stateimaging device such as a CCD or CMOS. One or a plurality of secondcameras 20 are attached to arbitrary locations on the host vehicle M.The second camera 20 may be provided at the same position as that of thefirst camera 10 or may be provided at a part of an installation positionof the first camera 10 (for example, a position at which a forward sidefrom the host vehicle M is imaged). The second camera 20, for example,repeatedly images the surroundings of the host vehicle M periodically.The second camera 20 may be a stereo camera.

The HMI 30 presents various types of information to a user of the hostvehicle M and receives an input operation from the user. Examples of theuser include a driver who drives the host vehicle M and an occupant suchas a fellow occupant. In the following description, an “occupant” willbe used unless otherwise specified. The HMI 30 includes, for example, adisplay and a speaker as outputs that present various types ofinformation. The display displays an image under the control of an HMIcontroller 170, which will be described below, and the speaker outputssound under the control of the HMI controller 170. The HMI 30 includes atouch panel, switches, keys, a microphone, or the like as an input thatreceives an input from the occupant. Information received by the inputis output to the HMI controller 170.

The vehicle sensor 40 includes, for example, a vehicle speed sensor thatdetects a speed of the host vehicle M, an acceleration sensor(three-axis G sensor) that detects an acceleration, a yaw rate sensorthat detects a yaw rate (for example, a rotation angle speed around avertical axis passing through a point of a center of gravity of the hostvehicle M), and an orientation sensor that detects a direction of thehost vehicle M. The vehicle sensor 40 may include a position sensor thatacquires a position of the host vehicle M. The position sensor is, forexample, a sensor that acquires position information (longitude andlatitude information) from a global positioning system (GPS) device. Theposition sensor may be, for example, a sensor that acquires the positioninformation using a global navigation satellite system (GNSS) receiver.The GNSS receiver specifies the position of the host vehicle M on thebasis of a signal received from a GNSS satellite. The position of thehost vehicle M may be specified or complemented by an inertialnavigation system (INS) using outputs of other sensors. A result ofdetection of the vehicle sensor 40 is output to the driving controldevice 100.

The driving operator 80 includes, for example, various operators such asa steering wheel with which a driver performs a steering operation, anaccelerator pedal, a brake pedal, and a shift lever. An operationdetector that detects an amount of operation performed by the driver,for example, is attached to each operator of the driving operator 80.The operation detectors detect an amount of depression of theaccelerator pedal or the brake pedal, a position of the shift lever, asteering angle or steering torque of the steering wheel, and the like.The operation detector outputs a detection signal indicating a detectionresult to the driving control device 100 or one or both of the travelingforce output device 92, the brake device 94, and the steering device 96.

The travel traveling force output device 92 outputs a travel drivingforce (torque) for traveling of the host vehicle M to driving wheels.The travel traveling force output device 92 includes, for example, acombination of an internal combustion engine, an electric motor, atransmission, and the like, and a power electronic control unit (ECU)that controls these. The power ECU controls the above configurationaccording to information input from the driving control device 100 orinformation input from the driving operator 80.

The brake device 94 includes, for example, a brake caliper, a cylinderthat transfers hydraulic pressure to the brake caliper, an electricmotor that generates hydraulic pressure in the cylinder, and a brakeECU. The brake ECU controls the electric motor according to theinformation input from the driving control device 100 or the informationinput from the driving operator 80 so that a brake torque according to abraking operation is output to each wheel. The brake device 94 mayinclude a mechanism that transfers the hydraulic pressure generated byan operation with respect to the brake pedal included in the drivingoperator 80 to the cylinder via a master cylinder, as a backup. Thebrake device 94 is not limited to the configuration described above andmay be an electronically controlled hydraulic brake device that controlsan actuator according to the information input from the driving controldevice 100 and transfers the hydraulic pressure of the master cylinderto the cylinder.

The steering device 96 includes, for example, a steering ECU and anelectric motor. The electric motor, for example, changes directions ofsteerable wheels by causing a force to act on a rack and pinionmechanism. The steering ECU drives the electric motor according to theinformation input from the driving control device 100 or the informationinput from the driving operator 80 to change the directions of thesteerable wheels.

[Configuration of Driving Control Device 100]

The driving control device 100 includes, for example, a recognizer 120,a controller 140, a driving controller 160, the HMI controller 170, anda storage 180. The recognizer 120, the controller 140, the drivingcontroller 160, and the HMI controller 170 are each realized by ahardware processor such as a central processing unit (CPU) executing aprogram (software). Some or all of these components may be realized byhardware (circuit; including circuitry) such as a large scaleintegration (LSI), an application specific integrated circuit (ASIC), afield-programmable gate array (FPGA), or a graphics processing unit(GPU), or may be realized by software and hardware in cooperation. Theabove-described program may be stored in a storage device (a storagedevice having a non-transitory storage medium) such as an HDD or a flashmemory of the driving control device 100 in advance, or may be stored ina detachable storage medium such as a DVD, a CD-ROM, or memory card andinstalled in the storage device of the driving control device 100 whenthe storage medium (non-transitory storage medium) is mounted in a drivedevice, card slot, or the like.

The storage 180 may be realized by any of various storage devicesdescribed above, an electrically erasable programmable read only memory(EEPROM), a read only memory (ROM), a random access memory (RAM), or thelike. The storage 180 stores, for example, information necessary forexecution of various controls in the embodiment, programs, and variousother information.

[Recognizer]

The recognizer 120 includes a surroundings recognizer 122, for example.For example, the surroundings recognizer 122 performs sensor fusionprocessing or the like on detection results of one or both of the firstcamera 10 and the radar device 12 to recognize a surroundings situationof the host vehicle M. In this case, the surroundings recognizer 122performs coordinate transformation of the image obtained from the firstcamera 10 into a bird's-eye view coordinate system, and performs knownschemes (binarization processing, contour extraction processing, imageenhancement processing, feature amount extraction processing, patternmatching processing, and the like) on the basis of the transformedcoordinate system to recognize the surroundings situation of the hostvehicle M. The surroundings recognizer 122 may recognize a position,type, speed, and the like of an object around the host vehicle M. Theobject is, for example, another vehicle (for example, a surroundingvehicle present within a predetermined distance from the host vehicleM). The object may include a pedestrian, a bicycle, a road structure,and the like. The road structure includes, for example, a road sign,traffic light, curb, median, guardrail, fence, wall, and railroadcrossing. The object may include an obstacle that hinder the travelingof the host vehicle M.

The surroundings recognizer 122 recognizes the host vehicle M, andstates such as positions (relative position), speed, acceleration, andthe like of an object present around the host vehicle M. The position ofthe object is recognized, for example, as a position in an absolutecoordinate system (hereinafter referred to as a vehicle coordinatesystem) with a representative point (a center of gravity, a center of adrive shaft, or the like) of the host vehicle M as an origin, and usedfor control. The position of the object may be represented by arepresentative point such as a center of gravity, corner, or a distaledge in a traveling direction of the object, or may be represented by arepresented area. Examples of the speed include speed of the hostvehicle M and the other vehicle relative to a traveling direction (alongitudinal direction) of a lane in which the host vehicle M and theother vehicle travel (hereinafter referred to as longitudinal speed),and speed of the host vehicle M and the other vehicle relative to alateral direction of the lane (hereinafter referred to as lateralspeed). Examples of the “state” of the object may include accelerationor jerk of the object when the object is a mobile object such as anothervehicle, or a “behavior state” (for example, whether the object ischanging lanes or is about to change lanes).

The surroundings recognizer 122 recognizes road marking likes(hereinafter referred to as demarcation lines) that are present aroundthe host vehicle M. For example, the surroundings recognizer 122recognizes left and right demarcation lines that define a lane in whichthe host vehicle M travels (hereinafter referred to as a host vehiclelane). The surroundings recognizer 122 separately recognizes an adjacentlane on the left side of the host lane (hereinafter referred to as aleft lane) and an adjacent lane on the right side of the host lane(hereinafter referred to as a right lane) on the basis of the left andright demarcation lines.

[Controller]

The controller 140 controls the entire driving control device 100. Thecontroller 140 includes, for example, a contour extractor 142, ademarcation line extractor 144, an edge acquirer 146, a belonging lanedeterminer 148, and a target capturer 150.

The contour extractor 142 extracts edge points from the two-dimensionalimage captured by the second camera 20 through existing image analysisprocessing, and extracts a contour of the object on the two-dimensionalimage from a sequence of the extracted edge point. The two-dimensionalimage is an image obtained by expressing the image captured by thesecond camera 20 in a two-dimensional coordinate system of alongitudinal direction (an X-axis direction) and a lateral direction(Y-axis direction) as it is. For example, the contour extractor 142connects edge points existing within a predetermined distance to extracta contour. The contour extractor 142 may acquire color information foreach edge point through image analysis processing for a two-dimensionalimage, connect the edge points whose acquired colors are similar, andextract the contour. The contour extractor 142 may extract only acontour of another vehicle (an example of a target object) on the basisof a shape, size, position, or the like of the contour. The contourextractor 142 separates and extracts each object when a plurality ofobjects are present in the two-dimensional image. The demarcation lineextractor 144 extracts demarcation lines included in the two-dimensionalimage captured by the second camera 20. The demarcation line extractor144 may separately extract the left and right demarcation lines of thehost lane. The demarcation line extractor 144 acquires positioninformation (coordinate information) on the two-dimensional image of theextracted demarcation lines.

The edge acquirer 146 acquires a position (position information on thetwo-dimensional image) of edges of the target object included in thetwo-dimensional image captured by the second camera 20. The edges of thetarget object are, for example, left and right edges on thetwo-dimensional image. Here, when the target object is another vehicle,edges of the other vehicle are, for example, left and right groundcontact points of the other vehicle (points at which the other vehiclecomes in contact with the road) in the two-dimensional image.Hereinafter, description is continued using the other vehicle as anexample of the target object.

The belonging lane determiner 148 determines a lane to which the othervehicle belongs (a presence settlement lane of the other vehicle) on thebasis of position information of the demarcation line extracted by thedemarcation line extractor 144 and position information of the edges ofthe other vehicle acquired by the edge acquirer 146. The belonging lanedeterminer 148 may determine a lane to which the other vehicle does notbelong (a lane in which the other vehicle is settled not to be present)instead of (or in addition to) the lane to which the other vehiclebelongs.

The target capturer 150 captures the target vehicle on which the hostvehicle M performs driving control using the driving controller 160 onthe basis of a recognition result of the recognizer 120 or a result ofdetermining the lane to which the other vehicle belongs from thebelonging lane determiner 148 (relative lateral position information ofthe other vehicle viewed from the host vehicle M). The target vehicleis, for example, a forward vehicle when the host vehicle M tracks theforward vehicle at a predetermined inter-vehicle distance under drivingcontrol such as adaptive cruise control (ACC) of the driving controller160. The target capturer 150 may extract other vehicles that interferewith the traveling of the host vehicle M from among other vehicles thatare present around the host vehicle M. The other vehicles that interferewith the traveling of the host vehicle M are, for example, othervehicles that require change in driving control (speed control andsteering control) of the host vehicle M due to lane change between apreceding vehicle and the host vehicle M when the host vehicle M tracksthe forward vehicle, or other vehicles that are likely to collide withthe host vehicle M and require driving control to avoid the collision.Details of functions of the contour extractor 142, the demarcation lineextractor 144, the edge acquirer 146, the belonging lane determiner 148,and the target capturer 150 described above will be described below.

The driving controller 160 controls one or both of the speed andsteering of the host vehicle M on the basis of a recognition result ofthe recognizer 120 and information from the controller 140. For example,when the driving controller 160 executes predetermined driving control,the driving controller 160 generates a scheduled trajectory (a goaltrajectory) along which the host vehicle M will travel in future, on thebasis of the recognition result of the recognizer 120 or the informationfrom the controller 140, in order to cause the host vehicle M to travelaccording to content of the driving control. The scheduled trajectoryincludes, for example, a speed element. The driving controller 160controls the speed or steering of the host vehicle M so that the hostvehicle M travels along the generated scheduled trajectory. The drivingcontroller 160 includes, for example, a speed controller 162 and asteering controller 164. For example, when the execution of ACC isreceived according to an operation with respect to the HMI 30 by theoccupant, the speed controller 162 controls the traveling force outputdevice 92 or the brake device 94 to perform speed control such asacceleration or deceleration so that the host vehicle M travels whilemaintaining a predetermined inter-vehicle distance from a precedingvehicle traveling in the host lane on the basis of the scheduledtrajectory. The speed controller 162, for example, controls thetraveling force output device 92 or the brake device 94 to perform speedcontrol such as acceleration or deceleration so that the host vehicle Mdoes not contact with the other vehicle on the basis of a situation inwhich the other vehicle approaches the host vehicle M.

For example, when the steering controller 164 performs driving controlsuch as ACC or Lane Keeping Assistance System (LKAS) according to anoperation with respect to the HMI 30 by the occupant, the steeringcontroller 164 controls the steering device 96 on the basis of a resultof recognizing the host lane and the position of the host vehicle M, inorder to maintain traveling in the host lane on the basis of thescheduled trajectory. For example, when auto lane changing (ALC) isexecuted according to an operation with respect to the HMI 30 by theoccupant, the steering controller 164 controls the steering device 96 toperform lane change to an adjacent goal lane along the scheduledtrajectory on the basis of the recognition result of the recognizer 120.

Processing of the speed controller 162 and the steering controller 164is realized, for example, by a combination of feedforward control andfeedback control. As an example, the steering controller 164 executes acombination of feedforward control according to a curvature of the roadin front of the host vehicle M and feedback control based on deviationfrom the scheduled trajectory (goal trajectory).

The HMI controller 170 presents predetermined information to theoccupant through the HMI 30. The predetermined information includes, forexample, information on a state of the host vehicle M and information ondriving control. Examples of the information on the state of the hostvehicle M include speed, engine speed, and a shift position of the hostvehicle M. Examples of the information on driving control includeinformation controlled by the controller 140, information on whether ornot driving control such as ACC or ALC, or manual driving is performed,and information on a situation of driving control that is beingexecuted. The predetermined information may include informationirrelevant to traveling control of the host vehicle M, such as a TVprogram or content (for example, movie) stored in a storage medium suchas a DVD.

For example, the HMI controller 170 may generate an image including thepredetermined information described above, and cause a display of theHMI 30 to display the generated image, and may generate a soundindicating the predetermined information and cause the generated soundto be output from a speaker of the HMI 30. The HMI controller 170 mayoutput information received by the HMI 30 to the controller 140, thedriving controller 160, or the like.

[Control of Vehicle Control Device]

Hereinafter, details of control that is executed by the driving controldevice 100 will be described. FIG. 2 is a diagram illustrating anexample of control that is executed by the driving control device 100.Before the driving control device 100 corrects a result of recognizingthe surroundings of the host vehicle M through the sensor fusionprocessing using information output by the first camera 10 and the radardevice 12, the driving control device 100 performs, for example, adetermination of the lane to which the other vehicle belongs using atwo-dimensional image of the surroundings (particularly, a travelingdirection) of the host vehicle M captured by the second camera 20, andacquisition of a relative positional relationship between thedemarcation line and the other vehicle.

The driving control device 100 can capture another vehicle that is atarget tracked by the host vehicle M (hereinafter referred to as atarget vehicle), for example, during execution of the driving controlsuch as ACC on the basis of the result of determining the lane to whichthe other vehicle belongs, the relative positional relationship betweenthe demarcation line and the other vehicle, an object recognition resultthrough the sensor fusion processing, and the like, generate, forexample, speed of the host vehicle M on the basis of the above and thelike on the basis of information on the captured target vehicle (forexample, a position, relative distance, relative speed, target vehicledistance, and traveling direction of the target vehicle) or the like,and cause the host vehicle M to travel at the generated speed. Thus, inthe embodiment, it is possible to curb error and contradiction in arelative position between the other vehicle and the host lane, byspecifying the belonging lane or acquiring the relative positionalrelationship in a coordinate system of the two-dimensional image withoutperforming, for example, three-dimensional coordinate transformation forobtaining a bird's-eye view image, using the two-dimensional imagecaptured by the second camera 20, and it is possible to capture thetarget vehicle for performing driving control on the basis of a moreaccurate relative position. The driving control device 100 may performsteering control instead of (or in addition to) the speed control.

[Processing Flow]

Next, a flow of processing that is executed by the driving controldevice 100 according to the embodiment will be described. FIG. 3 is aflowchart illustrating an example of the flow of processing that isexecuted by the driving control device 100. In the followingdescription, processing when the host vehicle M executes driving controlof an ACC among processing that is executed by the driving controldevice 100 will be mainly described. The processing illustrated in FIG.3 may be repeatedly executed while the driving control is beingexecuted.

In the example of FIG. 3 , the recognizer 120 first recognizes thesurroundings situation including in front of the host vehicle M throughsensor fusion processing using outputs from the first camera 10 and theradar device 12 (step S100). The recognizer 120 determines whether ornot another vehicle is present around the host vehicle M (step S120).The other vehicle is an example of the target object, and morespecifically, is another vehicle that is traveling in front of the hostvehicle M. The forward side of the host vehicle M includes not only theforward side on the host lane but also the forward sides on the leftlane and the right lane.

FIG. 4 is a diagram illustrating an example of the surroundingssituation of the host vehicle M. In the example of FIG. 4 , a three-laneroad consisting of a lane L1, a left lane L2 of a host lane L1, and aright lane L3 of the host lane L1 is shown. In FIG. 4 , it is assumedthat the host vehicle M (not illustrated) is traveling in the lane L1(hereinafter referred to as a host lane L1). In the surroundingssituation illustrated in FIG. 4 , the recognizer 120 recognizes anothervehicle m1 traveling in front of the host vehicle M and in the host laneL1 and other vehicles m2 and m3 traveling in the left lane L2, as targetobjects present in front of the host vehicle M.

The controller 140 executes the processing of steps S142 to S154 foreach of the other vehicles m1 to m3 recognized by the recognizer 120repeatedly (as a loop) (step S140). The contour extractor 142 extracts acontour of the other vehicle included in the two-dimensional imagecaptured by the second camera 20 (step S142). Next, the contourextractor 142 sets an area surrounding the contour of the other vehicle(step S144).

FIG. 5 is a diagram illustrating the area surrounding the contour of theother vehicle included in the two-dimensional image. In an example ofFIG. 5 , an image IM10 captured by the second camera 20 in asurroundings situation of the host vehicle M corresponding to FIG. 4 isillustrated. The image IM10 is a two-dimensional image in which thecaptured image is expressed in an image coordinate system as it is(without three-dimensional coordinate transformation). It is assumedthat, in the image IM10 illustrated in FIG. 5 , the contours of theother vehicles m1 to m3 are extracted by the contour extractor 142. Inthe image IM10 illustrated in FIG. 5 , the contour extractor 142generates three areas (hereinafter referred to as bounding boxes) BB1 toBB3 surrounding the extracted contours of the other vehicles m1 to m3(step S144). The bounding boxes BB1 to BB3 are, for example, rectangularareas formed along an X-axis (a longitudinal direction) and a Y-axis (alateral direction) of the image IM10. The contour extractor 142 extractscoordinates (XY coordinates) of four corners of each of the boundingboxes BB1 to BB3 at two-dimensional coordinates of the image IM10.

Next, the demarcation line extractor 144 extracts a point sequence ofdemarcation lines that define the host lane L1 from the two-dimensionalimage captured by the second camera 20 (step S146). FIG. 6 is a diagramillustrating the extraction of the demarcation lines. In the example ofFIG. 6 , portions of the lanes L1 and L2, and the other vehicle m2traveling in the lane L2 in the image IM10 captured by the second camera20 are illustrated in an enlarged manner. The demarcation line extractor144, for example, scans a lower end of the bounding box BB2 of the othervehicle m2 in a lateral direction (a left and right direction or aY-axis direction) of the image IM10 and acquires a position (XYcoordinate position) on the left and right demarcation lines LL and LRthat define the host lane L1 at the time of contact with the demarcationlines LL and LR. Further, the demarcation line extractor 144 similarlyperforms scanning with shift of a predetermined pixel (for example, onepixel) in an up-down direction (an X-axis direction or a longitudinaldirection) with reference to a line of the lower end of the bounding boxBB2, to extract areas of the demarcation lines LL and LR on thetwo-dimensional image within a predetermined range (hereinafter referredto as a specific area SP) consisting of 2n+1 pixels in the up-downdirection. n is, for example, about 10 to 30 and is, preferably, 20, butmay be changed appropriately according to resolution of thetwo-dimensional image, a surrounding environment of the host vehicle M,and the like. A lateral width of the specific area SP (a range in theleft-right direction) corresponds to a lateral width of the image IM10.The demarcation line extractor 144 extracts the demarcation lines LL andLR by connecting point sequences of coordinates obtained from theabove-described specific area SP.

Referring back to FIG. 3 , the demarcation line extractor 144 thenperforms noise removal on the demarcation lines LL and LR extractedwithin the specific area SP (step S148). FIG. 7 is a diagramillustrating removal of noise from the demarcation lines. For example,the demarcation line extractor 144 generates a histogram obtained bycounting the number of pixels when pixels extracted as demarcation linesare projected onto the Y-axis in the specific area SP of the image IM10.The histogram may be obtained, for example, by counting the number ofpixels arranged in each predetermined width in a Y-axis direction. Thedemarcation line extractor 144 extracts a section having a continuouslength in the Y-axis direction (a lateral direction) equal to or largerthan a predetermined length or a section in which the number of pixelsis equal to or larger than a threshold value in the generated histogram,as a section having a demarcation line. This makes it possible toprevent noise NS1 and NS2 such as extraneous matter included in theimage from being recognized as demarcation lines, for example, asillustrated in FIG. 7 , and to more accurately extract the demarcationlines.

Referring back to FIG. 3 , the demarcation line extractor 144 thenextracts an inclination of the demarcation line (step S150). Theinclination of the demarcation line is, for example, an angle formedbetween a lower end of the two-dimensional image captured by the secondcamera 20 and the demarcation line included in the two-dimensionalimage. For example, the demarcation line extractor 144 derives a firstprincipal component through principal component analysis processing forthe demarcation lines LL and LR in the specific area SP. FIG. 8 is adiagram illustrating extraction of the inclination of the demarcationline. In an example of FIG. 8 , the demarcation line LL extracted fromthe specific area SP for the other vehicle m2 is illustrated in anenlarged manner. In an example of FIG. 8 , a pixel group (a collectionof pixel data) extracted as the demarcation line LL is shown.

The demarcation line extractor 144 performs, for example, existingprincipal component analysis processing on the pixel group extracted asthe demarcation line LL to extract, for example, a direction in which avariance of a principal component score is the largest as an axis C1 ofthe first principal component. For example, the demarcation lineextractor 144 obtains a covariance matrix for coordinates at whichpixels corresponding to the demarcation line LL are present within apredetermined image area such as the specific area SP as illustrated inFIG. 8 , derives an eigenvector of the covariance matrix, and extracts adirection in which a variance (eigenvalue) is maximized, as the axis C1of the first principal component, from the derived eigenvector.

The demarcation line extractor 144 may extract an axis perpendicular tothe axis C1 of the first principal component as an axis C2 of a secondprincipal component. The demarcation line extractor 144 extracts theaxis C1 of the first principal component with respect to the Y-axis asan inclination of the demarcation line LL. The inclination isrepresented, for example, by the angle θ1 formed by the Y-axis and theaxis C1 of the first principal component. The demarcation line extractor144 similarly extracts, as an inclination of the demarcation line LR, anangle θ2 (not illustrated) formed between a Y-axis (more specifically, a−Y-axis) and an axis C1# (not illustrated) of the first principalcomponent of the demarcation line LR through principal componentanalysis, for the other demarcation line LR defining the host lane L1.As described above, the demarcation line extractor 144 acquires, asinclinations of the demarcation lines LL and LR, angles when demarcationlines of the specific area SP (an example of the predetermined area)with reference to a position of the demarcation line that meets when thelower end of the bounding box BB2 extends in a lateral direction of thetwo-dimensional image are viewed from a left-right direction (a Y-axisdirection or a lateral direction) of the two-dimensional image.

Referring back to FIG. 3 , the edge acquirer 146 then acquires positionsof left and right edges of the other vehicle present in thetwo-dimensional image (step S152). Next, the belonging lane determiner148 determines a lane in which the other vehicle travels (the lane towhich the other vehicle belongs) on the basis of a positionalrelationship between the left and right edges acquired by the edgeacquirer 146 and the demarcation lines (step S154). Next, the targetcapturer 150 performs capturing of another vehicle (target vehicle) thatis a target of the driving control executed by the host vehicle M (stepS160). Details of processing of steps S152 to S154 and S160 describedabove will be described below.

Next, the driving controller 160 executes driving control (either orboth of the speed control and the steering control) of the host vehicleM based on the surroundings situation and a behavior of the capturedtarget vehicle (step S170). In the processing of step S120, when thereare no other vehicles around the host vehicle M, the driving controller160 executes driving control of the host vehicle M based on thesurroundings situation (step S172). Thus, the processing of the presentflowchart ends.

[Acquisition of Left and Right Edges of Other Vehicle (Step S152)]

Next, processing of step S152 will be specifically described.

FIG. 9 is a diagram illustrating acquisition of left and right edges ofthe other vehicle. In an example of FIG. 9 , acquisition of the left andright edges of the other vehicle m2 will be described, but the sameprocessing is performed for other vehicles (the other vehicles m1 andm3) included in the two-dimensional image.

The edge acquirer 146, for example, performs scanning in an obliquedirection at an angle serving as a predetermined angle toward the topfrom the left and right lower ends of the image IM10 captured by thesecond camera 20 with the inner side of the image IM10 as a center toacquire positions in contact with the contour of the vehicle m2 as edgesof the other vehicle m2. The edge acquirer 146 may perform scanning inthe above-described oblique direction up to positions at which there arepixels corresponding to the other vehicle m2 (for example, contourpixels of the other vehicle m2) from left and right lower edges of thebounding box BB2 surrounding the contour of the other vehicle m2, ratherthan from the left and right lower edges of the entire image IM10, toacquire the positions in contact with the contour of the other vehiclem2 as the edges of the other vehicle. It is possible to reduce aprocessing load as compared to scanning from the entire image IM10 byperforming scanning within the bounding box BB2.

Here, for example, the inclinations θ1 and θ2 of the demarcation linesLR and LL that are angles between a lower end of the image IM10 capturedby the second camera 20 and the demarcation lines included in the imageIM are used as the predetermined angle (scanning angle) at which thescanning is performed. For example, when the edge acquirer 146 performsscanning from a lower right end of the bounding box BB2, the edgeacquirer 146 performs scanning at an angle serving as an inclination θ1of the demarcation line LL toward the top with an inner side as a centerfrom the lower end of the bounding box BB2, and when the edge acquirer146 performs scanning from a lower left end of the bounding box BB2, theedge acquirer 146 performs scanning at an angle serving as aninclination θ2 of the demarcation line LR toward the top with an innerside as a center from the lower end of the bounding box BB2, asillustrated in FIG. 9 . The edge acquirer 146 performs scanning whileshifting in a direction away from the left and right lower edges at thesame angle, to acquire respective positions in contact with the image ofthe other vehicle m2 as the left edge Le and the right edge Re. Forscanning, for example, each edge may be scanned in the same direction,the edges may be scanned alternately in the same direction and anopposite direction while shifting in a direction away from the lowerend, or the edges may be scanned in a zigzag with reference to the sameangle, as illustrated in FIG. 9 . For example, the left edge Le is theleftmost point when the other vehicle m2 is viewed from above, and theright edge Re is the rightmost point when the other vehicle m2 is viewedfrom above.

Normally, the other vehicle travels in a direction in which thedemarcation lines that define the lane extend. Therefore, an inclinationof the other vehicle viewed from the host vehicle M is highly likely todepend on inclinations of demarcation lines of a traveling lane.Therefore, it is possible to more accurately acquire left and rightedges (ground contact points) with respect to the inclination of theother vehicle along the lane in the two-dimensional image by performingscanning at a scanning angle with reference to an inclination of thedemarcation lines of a travelling lane (that is, demarcation lines ofthe host lane) of another vehicle traveling in an adjacent lane. Inparticular, because a body of another vehicle that changes lanes from anadjacent lane to a host lane is directed toward the host lane, it iseasy to ascertain a ground contact point of a front wheel of the othervehicle.

When the edge acquirer 146 performs scanning with reference to an anglethat is a predetermined angle, the edge acquirer 146 may generatesegmentation areas by further dividing the rectangular area of thebounding box BB2 and refer to the generated segmentation areas in apredetermined order to acquire left and right edges. The segmentationarea is a rectangular area including one or more pixels.

FIG. 10 is a diagram illustrating acquisition of edges of anothervehicle using the segmentation areas. Although in an example of FIG. 10, a state of scanning from a lower right end of the bounding box BB2corresponding to the other vehicle m2 is shown, the same processing isperformed for scanning from a lower left end. The edge acquirer 146 setsa segmentation area SA within the bounding box BB2, and scans the imagein the set area in a zigzag manner with reference to the obliquedirection at the predetermined angle θ1 described above from a lowerright end. In the example of FIG. 10 , numbers in the segmentation areaSA indicate a scan order set on the basis of an inclination direction C1of the demarcation line LL. The edge acquirer 146 determines whether ornot an image of the other vehicle M is included in each segmentationarea, and acquires coordinates of the segmentation area in which theimage of the other vehicle is first included (for example, coordinatesof a center of the area) as coordinates of the right edge of the othervehicle M. In the example of FIG. 10 , it is shown that the edge of theother vehicle M has been detected in a 14th segmentation area after thescanning starts.

When the edge acquirer 146 determines whether or not the image of theother vehicle M is included in each segmentation area, the edge acquirer146 may determine using images including only necessary information,instead of determining using an original image as it is. For example,image information with a reduced number of bits such as 8 bits is usedinstead of full-color image information at the time of thedetermination. This makes it possible to reduce an amount of data usedfor determination processing, thereby reducing a processing load.

When the scanning angles θ1 and θ2 (angles of the demarcation lines withrespect to a left-right direction (a Y-axis direction or a lateraldirection) of the image IM10) exceed a threshold angle θth, the edgeacquirer 146 sets the predetermined angle to a fixed value. Thepredetermined angle in this case is, for example, about 40 to 50 degreesand is, preferably, 45 degrees. When an upper limit of the scanningangle is set in this way, it is possible to correctly acquire the rightand left edges (ground contact points) with respect to an attitude ofthe other vehicle, for example, even when the other vehicle travels on acurved road.

[Determination of Lane to which Other Vehicle Belongs (Step S154)]

Next, processing of step S154 will be specifically described. Forexample, the belonging lane determiner 148 compares relative positionsof the left and right ground contact points viewed from the demarcationlines LL and LR on the basis of positions of the demarcation lines LLand LR of the host lane L1 included in the two-dimensional imagecaptured by the second camera 20 and the left and right ground contactpoints of the other vehicle, and determines which lane the other vehiclebelongs to (or which lane the other vehicle does not belong to). Thebelonging lane determiner 148 may determine whether or not anothervehicle is entering the host lane. “Another vehicle is entering the hostlane” is, for example, a case in which at least one of the two left andright edges (ground contact points) of the other vehicle is present inthe host lane.

FIG. 11 is a diagram illustrating a determination of a lane to which theother vehicle belongs. Although, in an example of FIG. 11 , a scene inwhich a lane to which the other vehicle m2 belongs is determined will bedescribed, the same processing is performed on other surroundingvehicles (the other vehicles m1 and m3 illustrated in FIG. 4 ). In theexample of FIG. 11 , it is assumed that left and right ground contactpoints (a left ground contact point Le and a right ground contact pointRe) of the other vehicle m2 are acquired by the edge acquirer 146.

The belonging lane determiner 148 acquires relative positions LD1 andRD1 of the right ground contact point Re viewed from the left and rightdemarcation lines LL and LR that define the host lane L1 on thetwo-dimensional image. The belonging lane determiner 148 acquiresrelative positions LD2 and RD2 of the left ground contact point Leviewed from the demarcation lines LL and LR. For a reference position ofthe demarcation lines LL and LR, for example, a position of thedemarcation line at the shortest distance from the ground contact pointsLe and Re from areas of the demarcation lines LL and LR included in thespecific area SP with reference to respective positions of the left andright ground contact points Le and Re (an area of (2n+1) rows shifted byn pixels in an up-down direction with reference to the same height asthe edge) is acquired.

When a part of the demarcation line cannot be recognized due to thepresence of the other vehicle m2 on the demarcation line, the belonginglane determiner 148 may acquire a reference position of the demarcationline at a shortest distance from the above-described ground contactpoint using a virtual demarcation line obtained by connectingrecognizable demarcation lines to the front and rear (front and back) ofthe non-recognizable part of the demarcation line in a direction inwhich the host lane L1 extends, linearly or non-linearly according to ashape of the road or a shape of another recognizable demarcation line.

The belonging lane determiner 148 determines whether the relativepositions when the ground contact points Le and Re are viewed from thedemarcation lines LL and LR are positive or negative. For the relativeposition, for example, a right direction from a certain reference pointRP on the two-dimensional image captured by the second camera 20 ispositive, and a left direction is negative. The belonging lanedeterminer 148 may reverse positive and negative for left and rightdirections. The belonging lane determiner 148 may perform thedetermination of positive or negative on the basis of relative positionsof the demarcation lines LL and LR when viewed from the ground contactpoints Le and Re. In the example of FIG. 11 , when the right directionis set to positive and the left direction is set to negative, therelative position of the right ground contact point Re viewed from thedemarcation line LL is positive (+), the relative position of the leftground contact point Le is negative (−), and the relative positions ofthe left and right ground contact points Le and Re viewed from thedemarcation line LR are both negative (−). The belonging lane determiner148 may determine a relative position of the ground contact point viewedfrom the demarcation line when a size (for example, a length) of therelative positions LD1, RD1, LD2, and RD2 is equal to or greater than apredetermined length. This makes it possible to suppress frequent changein signs of positive and negative when the other vehicle is traveling ina state in which the ground contact points of the other vehicle are inthe vicinity of the demarcation lines.

Next, the belonging lane determiner 148 determines the lane to which theother vehicle m2 belongs on the basis of a positional relationship(relative positions) between the left and right ground contact points Leand Re and the left and right demarcation lines LL and LR. FIG. 12 is aflowchart illustrating an example of a flow of processing fordetermining a belonging lane. In the example of FIG. 12 , first, thebelonging lane determiner 148 determines whether or not the relativeposition of the right ground contact point Re viewed from the leftdemarcation line LL is negative (step S200). When a determination ismade that the relative position of the right ground contact point Reviewed from the left demarcation line LL is negative, a determination ismade that the other vehicle m2 is present in the left lane (step S202).When a determination is made that the relative position of the rightground contact point Re viewed from the left demarcation line LL is notnegative, the belonging lane determiner 148 determines whether or notthe relative position of the left ground contact point Le viewed fromthe right demarcation line LR is positive (step S204). When adetermination is made that the relative position of the left groundcontact point Le viewed from the right demarcation line LR is positive,the belonging lane determiner 148 specifies that the other vehicle m2 ispresent in the right lane (step S206).

When a determination is made that the relative position of the leftground contact point Le viewed from the right demarcation line LR is notpositive, the belonging lane determiner 148 executes conditiondetermination processing for the lane to which the other vehiclebelongs, using the left and right ground contact points Le and Re andthe left and right demarcation lines LL and LR (step S208). In thecondition determination processing of step S208, the belonging lanedeterminer 148 determines whether or not each of a plurality of presetdetermination conditions is satisfied, and determines that the othervehicle m2 belongs to a lane according to each condition when thecondition is satisfied. FIG. 13 is a diagram illustrating a plurality ofdetermination conditions. In an example of FIG. 13 , content of thecondition and a belonging lane when the condition is satisfied areassociated with condition types 1 to 8. Types and content of conditiontypes are not limited thereto.

In the example of FIG. 13 , when the relative position of the leftground contact point Le viewed from the left demarcation line LL ispositive and the relative position of the left ground contact point Leviewed from the right demarcation line LR is negative (when thecondition of condition type 1 is satisfied), the belonging lanedeterminer 148 determines that the other vehicle m2 belongs to the hostlane. When the relative position of the right ground contact point Reviewed from the left demarcation line LL is positive and the relativeposition of the right ground contact point Re viewed from the rightdemarcation line LR is negative (when the condition of condition type 2is satisfied), the belonging lane determiner 148 determines that thatthe other vehicle m2 belongs to the host lane. When the relativeposition of the left ground contact point Le viewed from the leftdemarcation line LL is negative (when the condition of condition type 3is satisfied), the belonging lane determiner 148 determines that theother vehicle m2 belongs to the left lane. When the relative position ofthe right ground contact point Re viewed from the right demarcation lineLR is positive (when the condition of condition type 4 is satisfied),the belonging lane determiner 148 determines that the other vehicle m2belongs to the right lane.

When the relative position of the left ground contact point Le viewedfrom the left demarcation line LL is negative and the relative positionof the right ground contact point Re viewed from the left demarcationline LL is positive (when the condition of condition type 5 issatisfied), the belonging lane determiner 148 determines that the othervehicle m2 belongs to the left lane and the host lane. When the relativeposition of the left ground contact point Le viewed from the rightdemarcation line LR is negative and the relative position of the rightground contact point Re viewed from the right demarcation line LR ispositive (when the condition of condition type 6 is satisfied), thebelonging lane determiner 148 determines that the other vehicle m2belongs to the right lane and the host lane. When the relative positionof the left ground contact point Le viewed from the left demarcationline LL is positive and the relative position of the right groundcontact point Re viewed from the right demarcation line LR is negative(when the condition of condition type 7 is satisfied), the belonginglane determiner 148 determines that the other vehicle m2 belongs to thehost lane. When the relative position of the right ground contact pointLe viewed from the left demarcation line LL is positive and the relativeposition of the left ground contact point Le viewed from the rightdemarcation line LR is positive (when the condition of condition type 8is satisfied), the belonging lane determiner 148 determines that thatthe other vehicle m2 belongs to the host lane.

After the processing of step S208 ends, the belonging lane determiner148 merges (OR processing or logical sum operation) determinationresults based on the respective conditions (step S210), and determineswhether or not the other vehicle m2 belongs to both the right lane andthe left lane as a merging result (step S212). When the belonging lanedeterminer 148 determines that the other vehicle m2 belongs to both theright lane and the left lane, the belonging lane determiner 148determines that the other vehicle m2 belongs to the left lane, the hostlane, and the right lane (step S214). Thus, the present flowchart ends.

FIG. 14 is a diagram illustrating content of the processing of stepsS212 to S214. In an example of FIG. 14 , a presence settlement flag “1”is set for the lane to which the other vehicle is determined to belongwhen the respective conditions of condition type 3 and condition type 4are satisfied. For example, when the presence settlement flag is alsoset in both the left and right lanes for one other vehicle according tothe condition determination of condition type 3 and condition type 4,the belonging lane determiner 148 determines that the other vehicle isalso present in the host vehicle and sets “1” in a presence settlementflag of the host lane. This makes it possible for the belonging lanedeterminer 148 to recognize that the other vehicle belongs to (ispresent in) the host lane and the left and right lanes. For the flag, aflag such as a character such as “YES” or “O”, or a mark may be setinstead of the flag of “1”.

Although the belonging lane determiner 148 has determined the lane towhich the other vehicle m2 belongs (a presence settlement lane), thebelonging lane determiner 148 may determine a lane to which the othervehicle m2 does not belong (an absence settlement lane) instead of (orin addition to) such a determination. In this case, the belonging lanedeterminer 148 makes a determination on the basis of a predetermineddetermination condition for specifying an absence settlement lane. FIG.15 is a diagram illustrating processing for determining the absencesettlement lane. In an example of FIG. 15 , condition content and anabsence lane are associated with condition types A to D for specifyingabsence settlement lane. Types and content of condition types are notlimited thereto.

In the example of FIG. 15 , when the relative position of the rightground contact point Re viewed from the left demarcation line LL isnegative (when the condition of condition type A is satisfied), thebelonging lane determiner 148 determines that the other vehicle m2 isabsent in (does not belong to) the host lane and the right lane. Whenthe relative position of the left ground contact point Le viewed fromthe right demarcation line LR is positive (when the condition ofcondition type B is satisfied), the belonging lane determiner 148determines that the other vehicle m2 is absent in the host lane and theleft lane. When the relative position of the left ground contact pointLe viewed from the left demarcation line LL is positive (when thecondition of condition type C is satisfied), the belonging lanedeterminer 148 determines that the other vehicle m2 is absent in theleft lane. When the relative position of the right ground contact pointRe viewed from the right demarcation line LR is negative (when thecondition of condition D is satisfied), the belonging lane determiner148 determines that the other vehicle m2 is absent in the right lane.The belonging lane determiner 148 merges respective determinationresults based on conditions A to D to determine in which lane the othervehicle is absent.

The belonging lane determiner 148 may generate information in which aflag indicating a presence settlement lane and an absence settlementlane (absence lane) is set for each other vehicle for the host lane, theleft lane, and the right lane on the basis of the determination resultof the belonging lane for the other vehicle m2. FIG. 16 is a diagramillustrating an example of information in which a flag is set. Theinformation illustrated in FIG. 16 is an example of the relative lateralposition information of the other vehicle viewed from the host vehicleM, and is an example of relative position information of the othervehicle and the host vehicle M.

In the example of FIG. 16 , an example of a flag set when a positionalrelationship in the two-dimensional image (planar image) between theother vehicle m2 and the demarcation lines LL and LR that define thehost lane is the relationship illustrated in FIG. 11 is shown. In theexample of FIG. 16 , for each of the left lane, the host lane, and theright lane, a flag “1” is set in a lane in which the presence of theother vehicle m2 is settled and a lane in which the absence of the othervehicle m2 is settled. According to the example of FIG. 16 , it ispossible to specify that the other vehicle m2 is present in the leftlane and the host lane, but is not present in the right lane, accordingto the processing of the belonging lane determiner 148. For example,when it is found that the other vehicle m2 is not present in the hostlane as the absence settlement lane, this can be used for processingsuch as target vehicle selection or speed control in subsequent drivingcontrol such as tracking control. In the example of FIG. 16 , differentflags may be set for the presence settlement flag and the absencesettlement flag.

The belonging lane determiner 148 can determine one or both of a lane towhich the other vehicle belongs (presence settlement lane) and a lane towhich the other vehicle does not belong (absence settlement lane), forexample, even when one or a part of the left and right demarcation linesLL and LR that define the host lane cannot be recognized from thetwo-dimensional image captured by the second camera 20. The demarcationline cannot be recognized, for example, when extraction of edges of thedemarcation line cannot be performed from the two-dimensional image.Hereinafter, several determination patterns for the belonging lane in asituation in which the demarcation lines cannot be recognized will bedescribed.

<First Determination Pattern>

FIG. 17 is a diagram illustrating a first determination pattern.

The first determination pattern is a determination pattern fordetermining a presence settlement lane (affiliated lane) of the othervehicle m2 when the right demarcation line LR among the left and rightdemarcation lines LL and LR that define the host lane L1 cannot berecognized. In the first determination pattern, the belonging lanedeterminer 148 determines the presence settlement lane of the othervehicle m2 on the basis of the respective relative positions of theright ground contact point Re and the left ground contact point Leviewed from the left demarcation line LL. In the example of FIG. 17 , itcan be seen that, because the relative position of the right groundcontact point Re viewed from the left demarcation line LL is positiveand the relative position of the left ground contact point Le viewedfrom the left demarcation line LL is negative, the ground contact pointsRe and Le are present to straddle the demarcation line LL (the groundcontact points (edges) are present on the left and right sides of theleft demarcation line LL). Therefore, in the first determinationpattern, the belonging lane determiner 148 cannot recognize whether ornot the other vehicle m2 belongs to the right lane, but determines thatthe other vehicle m2 belongs to at least the host lane L1 (morespecifically, the other vehicle m2 belongs to the host lane L1 and theleft lane).

<Second Determination Pattern>

FIG. 18 is a diagram illustrating a second determination pattern. Thesecond determination pattern is a determination pattern for determiningan absence settlement lane of the other vehicle m2 when the leftdemarcation line LL among the demarcation lines LL and LR that definethe host lane L1 cannot be recognized. In the second determinationpattern, the belonging lane determiner 148 determines the absencesettlement lane on the basis of the respective relative positions of theright ground contact point Re and the left ground contact point Leviewed from the right demarcation line LR. In the example of FIG. 18 ,the relative positions of the right ground contact point Re and the leftground contact point Le viewed from the right demarcation line LR areboth negative. Therefore, in the second determination pattern, thebelonging lane determiner 148 determines that the other vehicle m2 doesnot belong to the right lane (the right lane is the absence settlementlane).

<Third Determination Pattern>

FIG. 19 is a diagram illustrating a third determination pattern. Thethird determination pattern is a determination pattern for determiningone or both of the presence settlement lane and the absence settlementlane of the other vehicle m2 when all of one of the demarcation linesand part of the other cannot be recognized. In an example of FIG. 19 , ascene in which a part of the left demarcation lane LL is not recognized(only a part is recognized), in addition to the right demarcation lineLR being not recognized, is shown. For the left demarcation line LL, thedemarcation line LL included in a specific area in a horizontaldirection (a lateral direction of the image) of the right ground contactpoint Re is recognized, but the demarcation line LL included in aspecific area in the horizontal direction of the left ground contactpoint Le is not recognized. Therefore, a relative position between thedemarcation line LL and the left ground contact point Le cannot beacquired. In this case, the belonging lane determiner 148 determines thelane to which the other vehicle m2 belongs on the basis of only therelative position of the right ground contact point Re viewed from thedemarcation line LL.

In the example of FIG. 19 , the relative position of the right groundcontact point Re of the other vehicle m2 viewed from the demarcationline LL is negative. Therefore, the belonging lane determiner 148 candetermine that the other vehicle m2 belongs to the left lane (the leftlane is the presence settlement lane of the other vehicle m2). Thebelonging lane determiner 148 can also determine that the other vehiclem2 does not belong to the host lane L1 and the right lane (the host laneL1 and the right lane are absence settlement lanes of the other vehiclem2).

<Fourth Determination Pattern>

FIG. 20 is a diagram illustrating a fourth determination pattern. Thefourth determination pattern is a determination pattern for determiningthe presence settlement lane of the other vehicle m2 in a first scene inwhich a part of each of both the demarcation lines LL and LR cannot berecognized. In the example of FIG. 20 , it is assumed that the rightdemarcation line LR between the demarcation lines LL and LR can berecognized in a Y-axis direction (a left-right direction of thetwo-dimensional image) at the right ground contact point Re of the hostvehicle M, and only the left demarcation line LL can be recognized inthe Y-axis direction at the left ground contact point Le. In this case,the belonging lane determiner 148 determines that the other vehicle m2belongs to the host lane L1 (the host lane L1 is the presence settlementlane of the other vehicle m2) in the first scene as illustrated in FIG.20 because a body of the host vehicle M is present at the left and rightground contact points Le and Re.

FIG. 21 is a diagram illustrating a second scene in which a part of eachof both demarcation lines cannot be recognized. In the second scene ofthe fourth determination pattern, only the demarcation line LL betweenthe demarcation lines LL and LR can be recognized in the Y-axisdirection at the right ground contact point Re of the host vehicle M,and only the demarcation line LR can be recognized in the Y-axisdirection at the left ground contact point Le. In this case, thebelonging lane determiner 148 can determine that the other vehicle m2belongs to the host lane L1 even in the scene illustrated in FIG. 21because the body of the host vehicle M is present at the left and rightground contact points Le and Re. It is possible to ascertain a presencesettlement lane or an absence settlement lane of another vehicle througha determination based on a relative position using a recognized part ofthe lane even in a state in which a part of the lane is not recognized,as in the fourth determination pattern described above.

When the determination is performed on the basis of the first to fourthdetermination patterns described above, it is possible to determine oneor both of the lane to which the other vehicle belongs (the presencesettlement lane) or the lane to which the other vehicle does not belong(the absence settlement lane), for example, even when one side or a partof the demarcation line cannot be recognized (edge extraction cannot beperformed) due to bad weather, or the like. Using the processing fordetermining the belonging lane described above, it is possible tofurther improve the accuracy of the determination of the lane to whichthe other vehicle belongs.

[Target Capturing (Step S160)]

Next, details of target capturing processing will be described. FIG. 22is a diagram illustrating a function of the target capturer 150. Thetarget capturer 150 includes, for example, an object filter 152, acontrol target capturer 154, an interference likelihood target extractor156, and a verifier 158.

The object filter 152, for example, extracts other vehicles (an exampleof the target objects) present in three lanes including the host lane,the left lane, and the right lane among objects present around the hostvehicle M on the basis of information on objects around the host vehicleM obtained through the sensor fusion processing of the recognizer 120.For example, the object filter 152 converts an image captured by thefirst camera 10 into a bird's-eye view coordinate system(three-dimensional coordinate transformation), and extracts othervehicles present in the three lanes on the basis of positions, shapes,or the like of objects on the image.

When the host vehicle M executes the driving control such as ACC, thecontrol target capturer 154 captures, as the target vehicle (a trackingtarget vehicle), the other vehicle that the host vehicle M tracks fromamong the other vehicles extracted by the object filter 152. The othervehicle that the host vehicle M tracks is, for example, a precedingvehicle that is present on a scheduled trajectory along which the hostvehicle M travels or is likely to be present on the scheduled trajectoryin future. The control target capturer 154 captures, as a new targetvehicle, a preceding vehicle present in a lane that is a lane changedestination, for example, when the host vehicle M changes lanes.

The control target capturer 154 sets a control transition ratio in aswitching period of the target vehicle when the target vehicle isswitched due to lane change. FIG. 23 is a diagram illustrating a settingof the control transition ratio in the switching period of the targetvehicle. In the example of FIG. 23 , it is assumed that the host vehicleM travels in a middle lane L1 of a road, which includes three lanes L1to L3 that vehicles can travel in the same direction, at a speed VM,another vehicle m1 travels at a speed Vm1 in front of the host vehicleM, and another vehicle m2 travels at a speed Vm2 in the lane L2 on theleft side of the host lane L1 in front of the host vehicle M. In theexample of FIG. 23 , it is assumed that the host vehicle M travels whiletracking the other vehicle m1 under the driving control such as ACC.

In this situation, when the host vehicle M further performs lane change(for example, ALC) from the lane L1 to the lane L2, the control targetcapturer 154 switches the tracking target vehicle (the target vehicle)from the other vehicle m1 to the other vehicle m2. For example, thecontrol target capturer 154 sets the control transition ratio on thebasis of the relative positions or speeds of the tracking other vehiclesm1 and m2 at each position on a scheduled trajectory K1 until the hostvehicle M performs lane change from the lane L1 to the lane L2. Thismakes it possible to execute driving control that provides a smoothbehavior at the time of lane change, for example, by performingadjustment (correction) of content of control, such as speed orsteering, according to a ratio such as 70% for control of a behavior ofthe other vehicle m1 and 30% for control of a behavior of the othervehicle m2.

For the setting of the control transition ratio described above, sincethe target vehicle of the host vehicle M is switched from the othervehicle m1 to the other vehicle m2, for example, even when the othervehicle m2 enters between the host vehicle M and the other vehiclethrough lane change in a state in which the host vehicle M tracks theother vehicle m1, the control transition ratio described above is set inthe switching period, and the driving control based on the set ratio isexecuted.

The interference likelihood target extractor 156, for example, sets anarea within a predetermined distance in a traveling direction from areference position (for example, a distal edge or a center of gravity)of the host vehicle M, including the host lane and the left and rightlanes thereof, as an interference likelihood area, and extracts othervehicles present in the area as target vehicles likely to interfere. Thepredetermined distance may be a fixed distance or may be variably set onthe basis of a road shape, road type, speed of the host vehicle, or thelike. The interference likelihood target extractor 156 may extract apredetermined number of other vehicles from each of the left lane, thehost lane, and the right lane or may extract a predetermined number ofother vehicles from side close to the host vehicle when a sum of othervehicles in the three lanes exceeds a threshold value. Further, when thedriving controller 160 is executing ACC (tracking traveling in which thehost vehicle M tracks a preceding vehicle), the interference likelihoodtarget extractor 156 extracts a cut-in vehicle (another vehicle) thathas entered (cut into) the host lane from an adjacent lane between thepreceding vehicle and the host vehicle M as a target vehicle likely tointerfere with the host vehicle M. When the target vehicle likely tointerfere is extracted, it is possible to execute more appropriatedriving control, for example, while the driving controller 160 adjusting(correcting) the content of the control such as speed or steering sothat the host vehicle does not come in contact with the target vehicle.

The verifier 158 verifies whether or not the target vehicle is a correcttarget for driving control (speed control and driving control) of thehost vehicle M, on the basis of the target vehicle captured by thecontrol target capturer 154 and the target vehicle extracted by theinterference likelihood target extractor 156, and the result ofdetermining the lane to which the other vehicle belongs in the belonginglane determiner 148. The respective target vehicles (for example, theother vehicles m1 to m3) and the respective other vehicles (for example,the other vehicles m1 to m3) whose belonging lane is determined by thebelonging lane determiner 148 are associated according to, for example,a relative position, shape, or size from the host vehicle M.

For example, when the belonging lane determiner 148 determines that thetarget vehicle (the other vehicle) recognized as being present in thehost lane L1, which has been extracted as the control target or theinterference likelihood target, does not belong to the host lane L1 (oris present in another lane other than the host lane), the verifier 158determines that the target vehicle is an incorrect target for drivingcontrol and corrects the recognition result of the recognizer 120.Specifically, the verifier 158 excludes the other vehicle that does notbelong to the host lane L1 (or that is present in another lane otherthan the host lane) among the other vehicles extracted as targetvehicles on the basis of the recognition result of the recognizer 120,from the target vehicles (the vehicles that are targets for drivingcontrol). Thus, since the two-dimensional image has a smaller error in aposition of the target object as compared with the recognizer 120 thatrecognizes an object while using three-dimensional image conversion orthe like, it is possible to suppress excessive speed control or the likedue to erroneous recognition by excluding a target vehicle not necessaryfor driving control using a result of determining the lane to which theother vehicle belongs on the basis of on the two-dimensional image (therelative lateral position information of the other vehicle). Theverifier 158 may correct a position of a target vehicle that is anexclusion target to match the belonging lane determined by the belonginglane determiner 148 instead of excluding the target vehicle notnecessary for driving control.

When the belonging lane determiner 148 determines that the recognizedtarget vehicle (other vehicle) present in the host lane L1 belongs to(is present in) the host lane L1, the verifier 158 determines the targetvehicle to be a correct target for driving control and outputsinformation on the other vehicle that is the target (targetinformation).

The speed controller 162 performs the speed control of the host vehicleM on the basis of the target information output by the verifier 158. Forexample, the speed controller 162 calculates a tracking goal controlamount for tracking at an appropriate inter-vehicle distance for thetracking target on the basis of a state quantity of the target vehicle(a relative distance from the target vehicle, relative speed, goalinter-vehicle), or the like included in the target information. Thespeed controller 162 adjusts a control amount on the basis of aninterference state with the interference likelihood target included inthe target information not to exceed a G limit determined according toacceleration or deceleration of the host vehicle M in advance (a limitvalue of G in the front and rear of the host vehicle M obtained from thevehicle sensor 40). The speed controller 162 adjusts a tracking controlamount on the basis of information such as a behavior (position andspeed) of the interference likelihood target, for another vehicle thatperforms lane change into the host lane L1 or another vehicle thattravels in a state in which the other vehicle has partially entered thehost lane (an example of the interference likelihood target). When thereis an interference likelihood target that has not been captured as thecontrol target, the speed controller 162 may perform the adjustmentaccording to a state such as a behavior of the interference likelihoodtarget. The speed controller 162 generates a speed profile for causingthe host vehicle M to travel on the basis of the tracking control amountobtained by the adjustment or the like and a current speed controlamount of the host vehicle M, and executes the speed control of the hostvehicle M on the basis of the generated speed profile.

The steering controller 164 also executes driving control such as ALC orLKAS on the basis of the target information described above to preventcontact with other vehicles.

This makes it possible to perform appropriate correction on a positionaldeviation of the target vehicle due to, for example, a recognition errorof the object recognized on the basis of three-dimensional imageconversion for a camera image or outputs of a plurality of sensors. Itis possible to improve the accuracy of recognition of the trackingtarget vehicle or a cut-in vehicle. When the belonging lane determiner148 determines that a vehicle is not present in the host lane, it ispossible to exclude the vehicle from the tracking target vehicle or thecut-in vehicle and suppress excessive deceleration. This makes itpossible to perform vehicle control with little sense of discomfort tothe occupant.

The HMI controller 170 may generate an image on which the other vehiclerecognized as the target vehicle and the other vehicles excluded fromthe target vehicle can be distinguished, and cause the generated imageto be displayed on the display of the HMI 30. This makes it possible tomore accurately notify the occupant of the target vehicle for drivingcontrol.

As described above, according to the embodiment, the vehicle controldevice includes the second camera (an example of the imager) 20 thatimages the surroundings of the host vehicle M, the recognizer 120 thatrecognizes the surroundings situation of the host vehicle M, the drivingcontroller 160 that controls one or both of speed and steering of thehost vehicle M on the basis of the recognition result of the recognizer120, and the controller 140 that controls the driving controller 160 onthe basis of imaging content of the second camera 20, wherein thecontroller 140 sets relative position information with respect to thehost vehicle M for an object around the host vehicle m present on thetwo-dimensional image captured by the second camera 20, and corrects therecognition result of the recognizer 120 on the basis of the setrelative position information, thereby more accurately recognizing theposition of the object included in the image and performing the drivingcontrol of the host vehicle. Therefore, it is possible to furtherimprove traffic safety.

Specifically, according to the embodiment, it is possible to suppress apositional deviation of the object (a deviation of the belonging lane)that occurs when estimation is performed on the basis of athree-dimensional conversion image (bird's-eye view image) in therecognizer 120 or outputs of a plurality of sensors, by determining thebelonging lane on the basis of a positional relationship between thedemarcation line and the edge of the target object on thetwo-dimensional image captured by the second camera 20. According to theembodiment, it is possible to improve the accuracy of the determinationof the lane to which the other vehicle belongs, by determining the edgesto be the ground contact points when the target object is the othervehicle. According to the embodiment, it is possible to limit a searchrange on the image and reduce a processing cost by acquiring the groundcontact points that are left and right edges by referring to asegmentation image from rectangle information of the bounding box.According to the embodiment, it is possible to accurately recognize thebelonging lane even when another vehicle straddles the lane.

According to the embodiment, it is possible to more accurately correct aposition deviation of the object due to a recognition error of theobject position estimated on the basis of the three-dimensionaltransformation or the outputs of the plurality of sensors, by acquiringa relative positional relationship between the demarcation linesdefining the host lane in which the host vehicle M travels and theobject (that is, a relative positional relationship between the hostlane and the object) using the second camera 20 alone before correctionof the recognition result using sensor fusion, and correcting therecognition result on the basis of acquired content. According to theembodiment, it is possible to improve accuracy of capturing of thetracking target or the cut-in target, by capturing the tracking targetor the interference likelihood target (a cut-in target) and thenperforming verification of correctness on a capturing result on thebasis of relative positional relationship information. It is possible tosuppress execution of driving control such as excessive speed control byexcluding the target vehicle found not to be present in the host lanethrough the verification. Therefore, according to the embodiment, it ispossible to execute vehicle control with less discomfort by improvingthe accuracy of capturing of the tracking target or interferencelikelihood target.

The embodiment described above can be expressed as follows.

A vehicle control device

including:

a storage device that stores a program, and

a hardware processor, and

configured to

image surroundings of a host vehicle using an imager;

recognize a surroundings situation of the host vehicle;

execute driving control to control one or both of speed and steering ofthe host vehicle on the basis of a result of the recognition;

control the driving controller on the basis of imaging content of theimager; and

set relative position information with respect to the host vehicle foran object around the host vehicle present on a two-dimensional imagecaptured by the imager, and correct the recognition result on the basisof the set relative position information, by the hardware processorexecuting the program stored in the storage device.

Although the modes for carrying out the present invention have beendescribed above using the embodiments, the present invention is notlimited to these embodiments and various modifications and substitutionscan be made without departing from the gist of the present invention.

What is claimed is:
 1. A vehicle control device comprising: an imagerconfigured to image surroundings of a host vehicle; a recognizerconfigured to recognize a surroundings situation of the host vehicle; adriving controller configured to control one or both of speed andsteering of the host vehicle on the basis of a result of the recognitionof the recognizer; and a controller configured to control the drivingcontroller on the basis of imaging content of the imager, wherein thecontroller sets relative position information with respect to the hostvehicle for an object around the host vehicle present on atwo-dimensional image captured by the imager, and corrects therecognition result of the recognizer on the basis of the set relativeposition information.
 2. The vehicle control device according to claim1, wherein, when the driving controller executes tracking traveling inwhich the host vehicle tracks a preceding vehicle and the recognizerrecognizes a new tracking target vehicle for the host vehicle or acut-in vehicle cutting into the host lane in which the host vehicletravels, the controller corrects control content of the drivingcontroller for the tracking target vehicle or the cut-in vehicle on thebasis of the relative position information with respect to the trackingtarget vehicle or the cut-in vehicle.
 3. The vehicle control deviceaccording to claim 2, wherein, when a determination is made that thetracking target vehicle and the cut-in vehicle are not present in thehost lane on the basis of the relative position information even in acase in which the recognizer recognizes the tracking target vehicle orthe cut-in vehicle, the controller excludes the tracking target vehicleand the cut-in vehicle from targets of driving control of the drivingcontroller.
 4. The vehicle control device according to claim 2, whereinthe controller causes the driving controller to execute driving controlfor the tracking target vehicle or the cut-in vehicle on the basis ofthe corrected recognition result.
 5. A vehicle control methodcomprising, on a computer: imaging, by an imager, surroundings of a hostvehicle; recognizing a surroundings situation of the host vehicle;executing driving control to control one or both of speed and steeringof the host vehicle on the basis of a result of the recognition;controlling the driving controller on the basis of imaging content ofthe imager; and setting relative position information with respect tothe host vehicle for an object around the host vehicle present on atwo-dimensional image captured by the imager, and correcting therecognition result on the basis of the set relative positioninformation.
 6. A computer-readable non-transitory storage medium havinga program stored therein, the program causing a computer to: imagesurroundings of a host vehicle using an imager; recognize a surroundingssituation of the host vehicle; execute driving control to control one orboth of speed and steering of the host vehicle on the basis of a resultof the recognition; control the driving controller on the basis ofimaging content of the imager; and set relative position informationwith respect to the host vehicle for an object around the host vehiclepresent on a two-dimensional image captured by the imager, and correctthe recognition result on the basis of the set relative positioninformation.